Steps Required for Automated Image Analysis

1. Identifying spots on the filter image (Spot Segmentation)
2. Extracting a single filter from the GAL file (GAL Extraction)
3. Aligning the GAL spots with the filter spots (Pattern Matching)
4. Extracting the intensities values from the filter image (Data Extraction)

See also: Applying Fiducial Marks

Image Coordinate Systems For this work, all data is referenced using the image coordinate system. The upper left corner of an image is the origin (e.g. 0,0). +X is right, +Y is down.

Example Image

For this document I will be using this image as my filter sample:

Spot Segmentation

In order to identify the spots on the filter image I tried the following:

Remove areas outside the filter boundaries from the filter analysis.

• This was determined by using a acceptance radius = (ImageHeight / 2) * 0.25. Pixels out side this radius are ignored
• This is important since the edge of the disk creates high intensity artifacts in the filter image.
  

  ImageIOService io = new org.mbari.esp.imaging.services.impl.ImageIOServiceImpl();
  ImagePlus ip = io.read((new File('filter_3.tif').toURI().toURL()));
  (new org.mbari.esp.imaging.RemoveNonFilterAreaOperation(ip.getProcessor())).run();
  io.write(new File('filter_3_roi.png'));

Apply a thresholding technique to locate spots.

• From testing a variety of thresholding techiques, the maximum entropy threshold provided the most reliable spot resolution.
  

  ImagePlus ip; // see above code for generating ROI image
  GeneArrayAnalysisService gaas = new org.mbari.esp.imaging.services.impl.GeneArrayAnalysisServiceImpl01();
  ImagePlus mask = gaas.mask(ip);
  io.write(new File('filter_3_roi_masked.png'));

  Resolving Spots Issue #1: Spot Size Spots are expected to be rounghly uniform size. To remove outlier spots I run statistics on the spots sizes that appear in the mask and use 90% confidence interval. Spots with sizes out side this range are rejected unless they're size is within a few pixels of a spot that was within the acceptable range. For example, given a set of spot sizes of (20, 12, 11, 10, 9, 8) with accepted spots of (12, 11, 10, 9), 8 would be accepted since it's close to 9 but 20 would not.

  Resolving Spots Issue #2: Spots Blurring Together In areas of bright spots grouped togther, it can be difficult to separate individual spots as they blur together when masked. I tried a technique of dilating spots (actually makes the spots smaller) to resolve individual bright spots. Notice in the masked image above that some spots that are discernible to the eye were removed by the code because they blurred together, resulting in a spot with a size that's outside the confidence limits

  Resolving Spots Issue #3: Bright Image There are problems with the thresholding technique when the image is saturated. The image below illustrate the misidentification of spots; red dots indicate where the spots were detected.

  Resolving Spots Issue #4: Bright 'Noise'

Extracting Spot Locations

Once a mask has been created it's relatively easy to 'extract' the spot location. For my analysis, I searched each spot, as defined by the mask, for the brightest 3x3 pixel block within the spot and used that as the spot location. This is useful for attempting to apply point-matching algorithms.

ImagePlus ip; // see above code for generating ROI image
GeneArrayAnalysisService gaas = new org.mbari.esp.imaging.services.impl.GeneArrayAnalysisServiceImpl01();
/*
 * Key = unique, but arbitrary, spot ID 
 * Value = x,y location of upper left corner of mean block. z is mean intensity of 3x3 block
 */
Map<Integer, Point3D<Double>> pointMap = gaas.extractIntensities(ip, 3);

GAL Feature Extraction

About GAL Files

Gene-array Layout Files (GAL) define the spots to be printed onto the filter by a piezzo-electric printer. A single GAL file defines multiple filters.

ESP GAL files aren't real GALs! Our printer uses a custom file format. So, while we call them GAL's, since they serve that role, they aren't real GAL files

GeneArrayIOService io = new org.mbari.esp.imaging.services.impl.GeneArrayIOServiceImpl();
List<GALPoint> points = io.read((new File('my_gal.txt')).toURI().toURL());
BufferedImage image = PiezzoArrayUtil.arrayToColorGrid(points, true, 100);
ImageIO.write(image, "png", new File("my_gal.png"));

Isolating a Single Filter from the GAL points

With a little apriori knowledge, this is pretty simple. Since there is a relatively large spacing between filters, I just create a spot defining the upper left corner (e.g. minimum x and y value of all points), define a box of a certain size (in our case I'm using 12500 units) and remove everything outside the box.

List<GALPoint> allFilters = io.read((new File('my_gal.txt')).toURI().toURL());
List<GALPoint> oneFilter = PiezzoUtilities.filterSingleSpot(allFilters, 12500);

Pattern Matching

At this point we have 3 important things:

1. The original filter image
2. Location of a number of spots from the image
3. Location of spots for a single filter as defined in the GAL file.
   

There are a number of techniques for pattern matching. A few that I've explored are described below:

Log-polar transformation.

References

Image registration using log-polar mappings for recovery of large-scale similarity and projective transformations.pdf

Evaluation

At first brush, this seems the ideal technique for image registration. It handles rigid transformations involving scaling, rotation, and translation. In order to use this technique I generated an ideal image of the GAL points and attempted to align that with the filter image. For the test image, I was able modify the transform to accuratly align rotation and give relatively good translation. However I was never able to get the image to scale correctly. This is in part due to the similarity of the patterns used in the image. The log-polar transform would create a peak correlation on the wrong patterns between the image. It should also be noted that other test images produced widely varying results. So I do not consider the technique reliable, although with more time, it may be possible to improve upon it.

Initial Images

Log-polar Transform of Images NOTE The GAL points image was duplicated vertically to allow account for wrapping of rotation when the cross-correlation is performed

Fourier-Mellin Transform

This is essentially a log-polar transform in Fourier space. I did not have time to pursue it.

Robust Point Set Registration Using Mixture of Gaussians

References

http://code.google.com/p/gmmreg/

Evaluation

This is a point set registration technique, so instead of attempting to align images, we are trying to match to different sets of points. This technique allows for both rigid and non-rigid point-set registration, although, in our case, we are only interested in rigid registration. (Non-rigid registration warps the points to fit, which we do not want) I ran the open-source code on our data samples and did not have any luck. I also passed the data on to the author, Bing Jian of this technique. He also was not succesful with the registration. His comments were:

I took a look at the results and noticed the rigid registration result is not as good as the nonrigid one. Maybe it is because of the scaling issue.

Have you found any better solution yet? I think the symmetry presented in the data can cause the local minimum problem. Do you have any ground truth for this case? I guess I need to add a similarity transform to handle this case. BTW, I suggest using the point set with less points as model set for the gmmreg program, but it should not matter too much as both point sets are small.

Data Extraction

Without being able to align the filter image with the GAL points, it's impossible to:

1. Associate a spot with it's content
2. Accurately and reliable extract all spots of interest.

Here's a few examples of just trying to extract spots from an image:

The Good

The Bad

The Ugly

Summary

The only hurdle that needs to be overcome at this point is pattern matching/image registration

Here are the signficant hurdles for pattern matching/image regsitration:

1. Reliably extracting spots from the filter image
   • In my test images, I was only able to extract all spots in one image using thresholding techniques.
   • Another technique to try would be edge detection. This may be problematic in noisier images though.
2. Dealing with symmetry in the spot layout, this confounds most image registration techniques.
   • It would be great if I could get some key spots that I can reliably detect and differentiate from the other spots. For example, 2 spots that are near the edge of the disk. If I could extract those then, I would be able to easily register the GAL file with the image. As an example, look at the image above an notice the single spot near the edge of the disk. If we had 2 of those on that image it would make my life much simpler!! ALthough in order to use those I would need be able to extract the same 2 spots from the GAL file.

Applying Fiducial Marks for Image Analysis

If the ESP arrays had distinct fiducial marks it should be possible to locate these marks and use them for scaling and rotation of the GAL to match the array image. Ideally these fiducial's should be either the closest points to the center of mass of the or the farthest points from the center of mass.

Applying Fiducials.

• During analysis we extract the spots that are visible using a thresholding technique. This give an image like this:
  
• From this mask we can calculate a center of mass of the image. (The blue dot represents the center of mass)
  
• To easily locate fiducial it would be best if they were either the points nearest the center-of-mass or farthest from it. The red dots in the images below represent notional fiducial points.
  • Example of points nearest COM:
    
  • Example of points farthest from COM:
    

filter_3.png (image/png)
HistogramUtilitiesTest_maximumEntropySplitTest.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_2-160s.tif.jpg (image/jpeg)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_1-160s.tif.jpg (image/jpeg)
fitler_3_aoi.png (image/png)
fitler_3_aoi_masked.png (image/png)
PointUtilitiesTest-testPlot01.png (image/png)
ArrayToColorGridTest01-20090504105215-A1-small.png (image/png)
FilterSingleArrayTest02-20090504105215-GRID.png (image/png)
PointUtilitiesTest-testPlot02.png (image/png)
Image registration using log-polar mappings for recovery of large-scale similarity and projective transformations.pdf (application/pdf)
ImageRegistrationTest01-test01-bestR2.png (image/png)
ImageRegistrationTest01-test01-bestR2p.png (image/png)
ImageRegistrationTest01-I.png (image/png)
ImageRegistrationTest01-Ip.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_3-40s.tif.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_2.tif.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_2-40s.tif.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_1.tif.png (image/png)
GeneArrayAnalysisServiceImplTest-_imaging_test01_filter_1-40s.tif.png (image/png)
filter_3-center-of-mass.png (image/png)
filter_3-distal_fiducial.png (image/png)
filter_3_center_fiducial.png (image/png)